Conversation
sgl-project#31501 added plan-time `window_left` for the SWA paged prefill wrapper and gated it on `spec_info is None`, noting in the same commit that the index trim is request-granular and "exactness comes from plan-time window_left". DFlash verify therefore still runs with the window compiled out: every query in the block shares one left bound anchored at the block start, so a query at block offset j sees j keys that a window ending at its own position excludes. DFlash verify is a linear block with no tree mask (the worker passes custom_mask=None), so nothing else supplies the window. Whitelist DFLASH_VERIFY; EAGLE tree verify keeps the old behaviour. The existing `runner_dflash_verify_swa_chain` case did not catch this because its reference mask was built from the same request-granular rule as the code under test. Correct the reference to the per-query window, which turns that pre-existing case into a regression test for this fix: with prefix_lens=(3,5) and window=4 the last query of the second request sees two keys too many without the change.
SubSir
requested review from
Fridge003,
HaiShaw,
Qiaolin-Yu,
hebiao064,
ispobock and
merrymercy
as code owners
August 17, 2026 22:16
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
#31501 added plan-time
window_leftfor the SWA paged prefill wrapper and gated it onspec_info is None. The same commit noted, on the index trim right above it, that the trimis request-granular and "exactness comes from plan-time window_left".
DFlash verify therefore still runs with the window compiled out. The whole block shares one
left bound anchored at the block start, so the query at block offset
jattends tojkeysthat a window ending at its own position excludes. DFlash verify is a linear block with no
tree mask -- the worker passes
custom_mask=None-- so nothing else supplies the window.Modifications
flashinfer_backend.py: allow the plan-time window forSpecInputType.DFLASH_VERIFY.EAGLE tree verify keeps the previous behaviour.
speculative_target_verify_runner.py: correct the DFlash reference mask fromprefix_len - windowtoq_pos - window.The reference mask was built from the same request-granular rule as the code under test,
which is why the pre-existing
runner_dflash_verify_swa_chaincase passed. With thereference corrected, that case becomes a regression test for this fix: at
prefix_lens=(3, 5)andsliding_window_size=4, the last query of the second request seestwo keys too many without the backend change.
Accuracy Tests
Verify output changes on SWA layers whenever the committed prefix exceeds the window. The
window it removes is at most
draft_token_num - 1keys at the oldest end of the window, sothe effect is small in absolute terms; the point is that verify and decode now use the same
window definition.
Measured on H200 with a sliding-window DFlash draft (window 2047, block 16) on flashinfer,
24 gsm8k prompts at T=0, this branch against the same tree with the backend hunk reverted:
The short row is the control: below the window the two masks are elementwise equal, and the
arms agree to the digit, which is also what makes the long row readable -- at T=0 with fixed
prompts the pipeline is deterministic, so 0.009 is the effect and not noise. It is that small
because the keys at stake are at most 15 out of 2047, at the oldest end of the window.
test/registered/attention/unittests/swa/test_flashinfer.py::runner_dflash_verify_swa_chainfails before the backend change and passes after it.
Speed Tests and Profiling
None. Passing
window_left >= 0selects the flashinfer module with the window predicatecompiled in; no additional work per step.
Checklist
CI States
Latest PR Test (Base): ❌ Run #32075240918
Latest PR Test (Extra): ❌ Run #32075240770